## Appendix **A Data Collection Matrix and Data Source** Appendix A Data Collection Matrix and Data Source 1 **Evaluation Questions Doc.**  **Review Interviews**  **Obser****** **vations SEA**  **District Principal**  **K teacher**  **Test**

**admin. PreK**  **I. How have KEAs been developed and adopted by four states at the forefront of using KEAs?** 1. How were KEAs developed or chosen (e.g., timeline, participants, content decisions)? 1,2,9 1a
-e,g, h ,2c 12 a. How were various stakeholders engaged in selecting a KEA? 3,10 3 b. For what key purposes was the KEA selected or developed? 4 2 11 c. What key criteria were used in the selection of the KEA (e.g., cost, burden, use with certain populations, connection to a PreK or K–3 assessment)? 5 4, 18,19 d. Why did the state choose an existing KEA measure or decide to develop its own (or some components of its own)? 6
-8 4 2. What are the characteristics of the specific KEAs selected? 11,15 3 13 a. What skills and domains are assessed? 10 b. What evidence or types of information are collected (e.g., observation, direct assessments, parent report)? 15 11 2 a 3 4 3 c. Is use of the KEA mandatory or optional for districts? What factors led to that decision? 1f, 8 3. Was the KEA pilot
- and field
-tested? If yes, how, and what was learned about its validity and reliability? 7 4. How is the KEA funded (both the up
-front cost and the annual cost)? What is the overall and per
-student cost to implement the KEA? How will funding be sustained? 9 9a **II. How are KEAs being implemented by 12 school districts within four states at the forefront of using KEAs?**  5. What general information and training on the administration of the KEA are provided to teachers, administrators, and parents? 12 13,14 3,4,5 5,6,7 5,6 4,5 2 5
- 15 a. What types of training models are used? 13 14a 5 a 7 a 5 a b. How well prepared were people to administer the KEA? What additional support would have been helpful? 5c
-f 7b
-e 6a
-c 5b,cAppendix A Data Collection Matrix and Data Source 2 **Evaluation Questions Doc.**  **Review Interviews**  **Obser****** **vations SEA**  **District Principal**  **K teacher**  **Test**

**admin. PreK**  6. How are KEAs administered? 1 2 2 1 1,2 1 14
- 16 a. When are KEAs administered? 14 1 2 3 2 15, b. Who administers the KEAs, and what training and support did those individuals receive? 15 12 a, 14 2 b,5 3a,6,7 4,6 3a, 5 15 ,18 c. How much time do the KEAs require of district staff, school administrators, and teachers to implement? 9 3 d 4d d. Are families involved in providing information? If so, how? 12 b 2 c 3 b 4 b 3 b 15 e. How is technology used to support KEA administration and scoring? 16 12c, 17 b 2 d 3 c 4c 3 c 15 f. What strategies are used to help ensure that children with disabilities and English learners are accurately assessed? 17 18, 1 9 10, 11 11,12 9,10 8,9 14 g. What aspects of the KEA implementation are standardized across districts and schools, and what aspects involve local discretion? 13,15 1 5 6 8 15 7. How are KEA data shared with the state? (e.g., Are results added to a student longitudinal data system or another permanent student record?) 18 1 7 8 10 8 7 15 8. How, if at all, do states and districts monitor and evaluate the implementation of KEAs? 19 1e,16 7 9 7 6 16 **III. How do these four states and their districts and schools use KEA results to inform policy and practice?**  9. Who receives the results of the KEAs (e.g., parents, PreK educators, kindergarten teachers, principals and early childhood administrators, the SEA)? 20 22 14,20 4,15 13,16 13 4
a. What information do they receive about the results? 20 23 1 5a
-c 1 6 13,16 10 4a b. Who has access to results and in what form and on what schedule? 20 23 a -c 15a
-d 4 b 10. What supporting information or training do stakeholders receive to help them interpret or use the results? 21 23 d ,25 15d,e,17, 18a,19a, 20b 18,20 14 11 5 16
- 18Appendix A Data Collection Matrix and Data Source 3 **Evaluation Questions Doc.**  **Review Interviews**  **Observati ons**  **SEA District**  **Principal K**  **teacher Test** 

**admin. PreK**  11. How do various stakeholders use KEA results? 23,24,26 16,18,19, 20 17,19 15 12 a. How aware of KEAs are PreK programs and teachers (including Pre K, Head Start, community
- based PreK programs, child care)? Do KEAs inform PreK program improvement? 20a 1,3,6,7 b. How, if at all, do teachers use KEA results to inform their classroom practice? 1 9 1 9 15 c. How, if at all, do principals use KEA results to inform decisions at the school level (e.g., classroom assignment for individual children, budget allocations, curriculum, or professional development)? 1 8 1 7 d. How, if at all, do districts use KEA results to inform curriculum or professional development? 2 6 1 6 e. How, if at all, do states review and use KEA results by student demographic characteristics, school district, or by PreK program? 23,24 f. How, if at all, do states use KEA results to inform PreK policy, curriculum, or other programmatic decisions? 2 4 12. Are there any policies regarding how KEA data should not be used (e.g., determine promotion or prevent kindergarten entry in any way)? If so, how are these policies communicated and enforced? 2 7 2 1 21 17 14 17 **IV. What lessons did states, districts, and schools learn about KEA adoption, implementation, and uses?** 13. What lessons can states learn about KEA adoption? For example, what processes were helpful, and what challenges needed to be overcome? 5,6,30 14. What lessons can states, districts, and schools learn about KEA implementation? For example, what implementation challenges were reported by state, district, and school staff, and what efforts did staff undertake to overcome them? 20,21,30 12,13,24 13,14, 22,23, 24 1 1,12,20 15, 16 10 15. What lessons can states, districts, and schools learn about using KEA results to inform policy and practice? 28,29,30 22,23,24 24 1 8,19,20 15, 16,17 8, 9, 10

## Appendix **B Copy of Federal Register Notice**  **(Placeholder)**

## Appendix **C Consent Form** A ppendix C Consent Form 1 **Case Studies of the Implementation and Use of Kindergarten Entry Asse ssments**  **PARTICIPANT CONSENT FORM Purpose**  This study, Case Studies of the Implementation and Use of Kindergarten Entry Assessments (KEAs), is sponsored by the U.S. Department of Education and is being carried out by SRI International (SRI), an independent, nonprofit research institute, and Decision Information Resources, Inc. The purpose of the KEA implementation case studies is to document the processes, accomplishments, challenges, and solutions of 24 schools in four states . Findings will be shared with federal and state policymakers and the public
. As part of the study, we are inviting you to participate in an interview that will include questions about [ tailored to respondent: KEA selection, development, adoption, and characteristics; KEA implementation; use of KEA results; and lessons learned]. The interviews are designed to last [ tailored to respondent: 45 minutes with school staff, 60 minutes with district
-level respondents, and 90 minutes with state
-level respondents ] . The interviews are intended to provide information about the overall KEA p rocess and not to evaluate individuals’ capabilities or performance. **Risks and Discomfort** We do not know of any risks or discomforts associated with this interview . Your participation in the interview is voluntary. If you decide to participate, you may discontinue your participation in the interview at any time without penalty of any kind. **Benefits** While there is no direct benefit to you, your participation in the study will contribute to the identification of effective KEA implementation practices that can then be disseminated to the many states engaged in or interested in implementing KEAs and support the technical assistance efforts of the U.S. Department of Education. **Confidentiality** We will treat the information you supply in a confidential manner. Only research staff will have access to your interview responses. You will not be identified by name in any report, and data will be reported in a manner that does not reveal your identify. Data will be aggregated across states, districts, and schools. St ates, however, will be identified by name. We would like your consent to record the interview. Recordings will be kept in a secure location by SRI International and will not be accessed by anyone outside the study team . The audio recordings will be destroyed at the conclusion of the study. You can participate in the interview but decline to have it recorded. Additionally, if you elect to have the interview recorded, you may stop the recording at any time.A ppendix C Consent Form 2

As part of the study, the study team will share its findings with the U.S. Department of Education and other federal agencies. The results of the evaluation will be summarized at the state level as well as across all the states and districts participating in the study. A publicly available f inal report integrating and consolidating the results from all the research activities is scheduled to be available in February 2016. **More Information** If this explanation leaves you with any unanswered questions, please ask and obtain answers before signi ng below. If you have questions later, please call the Project Director, Dr. Tracy Huang, at 650
-859
- 2338. For questions about your rights as a human participating in this research, you may contact SRI’s Human Subjects Committee at 650
-859
- 2686 and referen ce HS case number **[XXXX]**
. **Informed Consent** By signing this form you are indicating that you have read and understood the information provided to you about your participation in this interview, agree to participate in this study, and received a signed and dated copy of this consent form. Please check one: ____I am willing to participate in this interview and be audio recorded. ____I am willing to participate in this interview and not be audio recorded. I understand that this is voluntary and is part of a study sponsored by the U.S. Department of Education and conducted by SRI International and Decision Information Resources, Inc.

_______________________________________ _______________________ Name (please print) Date

_______________________________________ __________________________________ Signature Job Title

## Appendix D **Recruitment Materials**Appendix D Recruitment Materials 1



## KEA **Case Studies Sample (Interview ) Recruitment Letter**  

# State	Recruiting	LetterAppendix D Recruitment Materials 2



## [Department of **Education letterhead]** Dear **[State Superintendent]** : The U.S. Department of Education is sponsoring a study to learn about the development, implementation, and use of Kindergarten Entry Assessments (KEAs). **[Insert name of state]** is one of four selected to participate. Each state’s case study will consist of document reviews, observations of one or two professional development sessions on the administration and use of KEAs, interviews with state - level staff, interviews with distr ict
- level staff in three districts, and site visits and interviews at two schools within each district. The result of this effort will be a report that identifies cross
-site lessons, revealing how practices work on the ground and implementation challenges and solutions. States are approaching KEAs in various ways and your state was chosen because the U.S. Department of Education believes that your experiences will help other states think through their decisions about which KEA to use, how to the KEA, and h ow to use the KEA results. SRI International and Decision Information Resources, Inc. (DIR) are conducting the study. A representative from SRI or DIR will contact **[Insert name of SEA representative]** , **[Insert role of representative]**  , soon to discuss the study and your state’s plan and schedule for KEA training and implementation. If you have questions, you may contact the Project Director, Dr. Tracy Huang, at 650
- 859
- 2338 or Erica Lee at the U.S. Department of Education at erica.lee@ed.gov. We appreciat e your cooperation in helping us gather and disseminate information about how states, districts, and schools are developing, implementing, and using KEAs in order to strengthen these efforts across the country. We look forward to learning more about the in teresting work that you are doing. Sincerely,
[ **Name** ], Director [ **Name** ], Deputy Assistant Secretary for Policy and Program Studies Service Policy and Early Learning Office of Planning, Evaluation and Policy Development Office of Elementary and Secondary Education
cc: [SEA representative(s)]Appendix D Recruitment Materials 3



## KEA **Case Studies Sample (Interview ) Recruitment Email**  

# State	Version	for	Key	Informants
Dear **[State representative]** , The U.S. Department of Education is sponsoring a study to learn about the development, implementation, and use of Kindergarten Entry Assessments (KEAs). [Insert name of state] is one of four selected to participate. **[Insert name of SEA contact]** identifie d you as a key informant in your state to talk with about your experiences with and lessons learned about the selection and use of a KEA. We are asking for your participation in a phone interview. This email tells you about the purpose of the interview and how the responses will be used. The study is being carried out by SRI International and Decision Information Resources, Inc. (DIR). Our study team is inviting state and district administrators as well as local educators to share with us what they have le arned about the processes, accomplishments, and challenges of implementing KEAs . We believe information about your experiences will be valuable for other states interested in implementing KEAs and for federal and state policy makers. The interview will ta ke approximately 90 minutes of your time. We can schedule it at your convenience during **[proposed time period]**
. The interviews are intended to provide information about the overall KEA system and not to evaluate individuals’ capabilities or performance. Qu estions cover topics such as the purpose of your state’s KEA, the reasons for choosing your state’s KEA, funding of the KEA, administration of the KEA, and uses of KEA results. Our study team will summarize our findings from your state’s case study as wel l as integrate and consolidate the results across all the states and districts in the study. We will present the results to the U.S. Department of Education and produce a public final report planned to be available in 2016. Your responses to the interview questions will be kept confidential, as names and other identifying information will not be included in any reports. Please let me know when you are available for an interview during [propose time period]. If you have questions about the interview, please email me or contact me at the number below . We appreciate your cooperation in helping us gather and disseminate information about how states, districts, and schools are developing, implementing, and using KEAs in order to strengthen these efforts across t he country. We look forward to learning more about the interesting work that you are doing. Respectfully, **[SRI Interviewer name and contact information]**Appendix D Recruitment Materials 4 

## Follow- up email reminder about scheduled interview Dear **[State representative]** , This is a friendly reminder about our 90
- minute phone interview about the selection, implementation, and use of your kindergarten entry assessment (KEA) . We are scheduled to talk on **[Day], [Date]** , at **[Time]** . I will call you at this number **[Phone Number]** un less there is a more convenient place to reach you. In preparation for the interview, I am sending a few sample questions and a list of documents that might be helpful to have on hand during the call. The sample questions are:

 For what key purposes was the KEA selected or developed in your state?  What were the primary reasons you chose the particular KEA(s) for your state? o How, if at all, does the KEA align with the domains of your state’s early learning standards?  How is the KEA currently funded? Is th ere a plan for sustaining future implementation?  How is the KEA administered across the state?  How are KEA results shared and used to inform policy and practice (at the state, district, and school levels)?  What are the most important lessons learned about KEA selection, implementation, and use of results? These are the documents that may be helpful to have on hand:

 Summary of the KEA selection process in your state  Pilot and/or field test findings  Map of KEA domains/skills to your state’s early learning standards  General information provided to parents, teachers, and administrators about the KEA  Example reports of KEA results. If these documents are available electronically, please send them to me before the interview. If you have any questions about the interview, please don’t hesitate to contact me. I look forward to our discussion. Sincerely, **[SRI Interviewer name and contact information]**Appendix D Recruitment Materials 5



## KEA **Case Studies Sample (Interview ) Recruitment Letter**  

# District	Recruiting	LetterAppendix D Recruitment Materials 6



## [Department of Education letterhead] Dear **[District Superintendent]** : The U.S. Department of Education is sponsoring a study to learn about the development, implementation, and use of Kindergarten Entry Assessments (KEAs). [Insert name of state] is one of only four in the United States selected to participate in case studies sponsored by the U.S. Department of Education to learn about the implementation and use of kindergarten entry assessments (KEAs). Further, **[Insert name of SEA contact]** identified your district as a good one to learn from about how KEA implementation works on the ground. The study is being carried out by SRI International and Decision Information Resources, Inc. (DIR). In each district, the study team will conduct phon e interviews with three district staff supporting KEA implementation. The team will also visit two schools per district to interview school administrators and teachers (six staff members per school), collect and review state and local documents, and possib ly observe one professional development session on how to administer the KEA or how to use the resulting data. The result of this effort will be a report that identifies cross
-site lessons, revealing how practices work on the ground and implementation chal lenges and solutions. A representative from SRI or DIR will contact you soon to discuss the study and your district’s plan for KEA implementation. If you have questions, you may contact the Project Director, Dr. Tracy Huang, at 650
-859
-2338 or Erica Lee a t the U.S. Department of Education at erica.lee@ed.gov. We appreciate your cooperation in helping us gather and disseminate information about how states, districts, and schools are developing, implementing, and using KEAs in order to strengthen these effo rts across the country. We look forward to learning more about the interesting work that you are doing. Sincerely,
[ [ **Name** ], Director [ **Name** ], Deputy Assistant Secretary for Policy and Program Studies Service Policy and Early Learning Office of P lanning, Evaluation and Policy Development Office of Elementary and Secondary Education **cc: [SEA representative] [District representative]** Appendix D Recruitment Materials 7



## KEA **Case Studies Sample (Interview ) Recruitment Email**  

# District	Version	for	Key	Informants
Dear **[District representative]** , The U.S. Department of Education is sponsoring a study to learn about the development, implementation, and use of Kindergarten Entry Assessments (KEAs). [Insert name of state] is one of four included in the study, and **[Insert n ame of district]**  is one of the three districts in [Insert name of state] selected to participate. **[District contact name], [role of district contact],** identified you as a key informant in your district to talk with about your experiences and lessons learned about the use of a KEA. We are asking for your participation in a phone interview. This email tells you about the purpose of the interview and how the responses will be used. The study is being carried out by SRI International and Decision Information Resources, Inc. (DIR). Our study team is inviting district administrators as well as local educators to share with us what they have learned about the processes, accomplishments, and challenges of implementing KEAs . We believe information about your experiences will be valuable for other districts interested in implementing KEAs and for federal and state policy makers. The interview will take approxima tely 60 minutes of your time. We can schedule it at your convenience during **[proposed time period]**
. The interviews are intended to provide information about the overall KEA system and not to evaluate individuals’ capabilities or performance. Questions cove r topics such as the administration of the KEA, the uses of KEA results, and lessons learned. Our study team will summarize findings from interviews conducted across your district and then integrate and consolidate results across your state with the fin dings from the three other states in the study. We will present the results to the U.S. Department of Education and other federal agencies, and we will produce a public final report planned to be available in February 2016. Your responses to the interview questions will be kept confidential, as names and other identifying information will not be included in any reports. Please let me know when you are available for an interview during [propose time period]. If you have questions about the interview, please email me or contact me at the number below . We appreciate your cooperation in helping us gather and disseminate information about how states, districts, and schools are developing, implementing, and using KEAs in order to strengthen these efforts across t he country. We look forward to learning more about the interesting work that you are doing. Respectfully, **[SRI Interviewer name and contact information]**Appendix D Recruitment Materials 8 

## Follow- up email reminder about scheduled interview Dear **[District representative]** , This is a friendly reminder about our 1
- hour phone interview about the implementation and use of your kindergarten entry assessment (KEA) . We are scheduled to talk on **[Day], [Date]** , at **[Time]** . I will call you at this number **[Phone Number]** unless there is a more convenient place to reach you. In preparation for the interview, I am sending a few sample questions and a list of documents that might be helpful to have on hand during the call. The sample questions are:

 How is the KEA administered in your district?  How are KEA results shared and used to inform policy and practice (at the state, district, and school levels)?  What are the most important lessons learned about KEA selection, implementation, and use of results? These are the documents that may be helpful to have on hand:

 General information provided to parents, teachers, and administrators about the KEA  Example reports of KEA results. If these documents are available electronically, please send them to me before or after the interview. If you have any questions about the interview, please don’t hesitate to contact me. I look forward to our discussion. Sincerely, **[SRI Interviewer name and contact information]**Appendix D Recruitment Materials 9



## KEA **Case Studies Sample (Interview ) Recruitment Email**  

# School	Recruiting	Letter
Dear **[School Principal]** : The U.S. Department of Education is sponsoring a study to learn about the development, implementation, and use of Kindergarten Entry Assessments (KEA**s). [Insert name of state]** is one of only four in the United States selected to participate in case studies sponsored by the U.S. Department of Education to learn about the implementation and use of kindergarten entry assessments (KEAs) and your district was identified as a good on e to learn from about how KEA implementation works on the ground. **[District contact]** identified your school as a good one to learn from about how KEA implementation works on the ground. The research study is being carried out by SRI International and Dec ision Information Resources, Inc. (DIR). In each district, the study team will visit two schools per district to conduct in
- person interviews. At each school, the will interview the principal, two kindergarten teachers, two PreK educators (e.g., teachers or director working with 4
-year-olds), and a test administrator or other staff member involved with KEA implementation (six staff members total). The study team will also collect and review state and local documents and possibly observe one professional development session on how to administer the KEA or how to use the resulting data. The result of this effort will be a report that identifies cross
-site lessons, revealing how practices work on the ground and implementation challen ges and solutions. We will contact you soon to discuss the study and logistics of the site visit. If you have questions, you may email me or Erica Lee at the U.S. Department of Education at erica.lee@ed.gov. We look forward to talking with you. Respectfully, **[SRI Interviewer name and contact information] Cc: District rep** 

## Appendix E **Document Review Protocol**Appendix E Document Review Protocol 1 

## KEA Case Studies 

# Document Review Protocol **Purpose** The purpose of this protocol is to review documents in a systematic way to provide background information about a state’s KEA for interviews and more technical or detailed information about some aspects of the processes used for selection, implementation, and reporting and use of results. The data will be used to  Provide background on a state’s KEA for state interviews and site visits on a state’s KEA development/adoption, implementation, and use for state and final report.  Gather more technical and detailed information about the KEA tool and process.  Abstract re levant information from documents obtained at interviews and observations. **Resources** Documents may be obtained from the following sources: a. Applications to Race
-to -the
-Top Early
- Learning Challenge b. State Department of Education or Department of Early Learni ng website, which contains the following types of resources:  KEA strategic plans or planning documentation  Pilot or field test report  KEA information for various stakeholders  KEA training materials  Assessment manuals  KEA reports with findings for various stakeholders. c. State, district, or school interview respondents who share documents with the interviewers. **General Instructions and Tips** a. Populate header information first so you do not forget. b. This is a cumulative document that will first be populated with information from publically available documents and later updated with documents provided to the study team by interview respondents. c. There will be one document review per state. All sources of information should be listed in the table on page 2, “Document Source for Information.” Each document source should be numbered sequentially and referenced in each section as the review is being conducted. d. Make notes of any questions you have when reviewing the documents. When unsure of the information, flag it and note the document name and page number, if available, and ask another team member to review. This will allow the interview team to identify areas in which clarification are needed. e. You are not limited to the format or the table presented. Add any notes as needed.Appendix E Document Review Protocol 2 

### Document Source for Information **Source # Title**  **Date Where document came**  **from If website, list URL**  **Author** 1. 2. 3. 4. 5. 6. **Add rows as needed List sources of information used (by source number) next to each item below.**  **Section I: Background on KEA Development or Adoption** 1. What year did the state begin the initiative to implement the KEA? **(Q1)** 2. What agency or department was the key driver of the KEA initiative? List any collaborations with another agency or department during the early stages of the KEA initiative. **(Q1)** 3. What other stakeholders (such as district representatives, Kindergarten teachers, PreK teachers, parents, foundation representati ves, outside contractors/expert consultants) were engaged in the KEA selection process and how? **(Q1a)** 4. For what key purposes was the KEA selected or developed? [Check all that apply.] **(Q1b) Check**  **Key purposes Source #**  Screening: To identify children who may need additional supports or further assessment Data
- informed instruction: To identify specific skills to teach to individual students or groups of students Family engagement: To inform parents about children’s strengths and needs and ways to support learning Professional development: To inform professional development for teachers about a common student need Formative assessment: To benchmark learning in order to track progress throughout the year EC program improvement: To inform improvement of early learning programs/policies Other (specify):Appendix E Document Review Protocol 3

5. What criteria were used in the selection of the KEA? [Check all that apply.] **(Q1c) Check**  **Criteria Source #**  Cost Match to early learning standards Burden Use with certain populations Connection to a PreK assessment Connection to a K
-3 assessment Other (specify): 6. Which of the following KEA measures was selected? [Check all that apply.] **(Q1d) Check**  **KEA measures Source #**  Teaching Strategies
-Gold Desire Results Developmental Profile
-School Readiness Work Sampling BRIGANCE DIBELS Other (specify): Other (specify): 7. How was the above measure modified, if at all, to better fit the state’s needs? **(Q1d)**
8. If state developed its own KEA measure, what was the main reason for doing so? **(Q1d)**
9. What agency or department is currently overseeing the implementation of KEA? **(Q1)**
10. Who was involved in selection of KEA measure? [Check all that apply.] **(Q1a) Check**  **People/agencies involved Source #**  State department leaders District representatives Teachers Parents Outside contractors (specify): Experts (specify): Foundation representatives (specify): Other (specify):Appendix E Document Review Protocol 4

11. What are the domains assessed by the KEA measure? [Check all that apply.] **(Q2) Check**  **Domains Source #**  Language and literacy Cognition and general knowledge, including math and early science Approaches to learning Physical well
- being, including motor development Social and emotional development Other (specify): Other (specify): Other (specify): Other (specify): **Section II: KEA Implementation** 12. What information on the administration of the KEA is provided to various stakeholders ? **(Q5) Stakeholders**  **Topics covered Source #**  District administrators Principals K teachers PreK teachers Parents Other (specify): 13. What training or support do teachers and administrators receive on the administration of the KEA ? [Check all that apply.] **(Q5a)** Is the training/support offered the same for **all** schools and districts statewide or is it school or district specific? **(Q6g)**

**Check Type and source of** 

**training/support How often**  **it is offered How long** 

**the training is**  **Specific to** 

**district or same**  **statewide Source #**  In
- person training by: 1
-on - 1 coaching by: TA by: Webinars by: Online self
- paced courses or modules List URL: OtherAppendix E Document Review Protocol 5

14. When is the KEA administered (e.g., within what week of school)? **(Q6a)** First assessment:

__________________________ __________________________ Second assessment (if applicable):

_______ _______________________________ Third assessment (if applicable):

_________ _______________________________ Any other administrations:

________________ ____________________________ 15. Who administers or provides information used in the KEA? Also indicate what evidence or types of information are gathered. [Check all that apply.] **(Q6b) (Q2,2b)**  Are the procedures the same for all schools and districts statewid e or is it school or district specific? **(Q6g) Use multiple rows for each skill/domain, depending on who collects the data and what**  **information is collected.**

**Skill/ Domain**  **Who**

**collects? (e.g.,**  **teachers, classroom**  **aids, families,**  **others) What information is collected? (e.g.,**  **checklists, direct assessments,**  **observation ratings) Specific**  **to district or**  **same statewide**  **Source #**

16. How is technology used, if at all, to support KEA administration and scoring? [Check all that apply.] **(Q6e)**
**Check if used**  Source #

Collect documentation for portfolios Provide direct assessment tasks online Enter data on ratings and student information Score data collected by teachers or others Generate reports on results Other (specify):Appendix E Document Review Protocol 6

17. What strategies are used to help ensure that children with disabilities and English learners are accurately assessed? [Check all that apply.] **(Q6f)**
**Strategies used Children**  **with disabilities**  **English**

**Learners Source #**  State offers special training and supp ort to those administering the KEA with CWD and/or ELs Offer alternate assessments for CWD Offer additional accommodations during administration of KEA for CWD Offer support to conduct assessment in the child’s native language Provide modified or adapted results Other (specify):

18. How do schools and districts submit KEA data/results to the state? [Check all that apply.] **(Q7)**
**Check if used**  **Specific to**  **district or same**  **statewide Source #**  Paper forms Exports Data entry into special KEA data system Data entry into SLDS Other (specify):
19. List how the state or districts monitor and evaluate implementation of the KEA. **(Q8)**
 Xx
 XxAppendix E Document Review Protocol 7

**Section III: Use of KEA** 20. Who receives or has access to the results of the KEAs? [Check all that apply.] **(Q9,9a,9b)**

**Check if used**  **Who has access**

**to the results? In what format** 

**(e.g., printed reports, web data**  **system)? When?**  **What type of information?**  **Source**

**#** Families PreK educators K teachers Principals District administrators SEA administrators Other (specify) 21. What supporting information or training do teachers, principals, district staff, and others receive regarding interpreting or using KEA results? [Check all that apply.] **(Q10)**

**Check if used**  **Staff Type of**  **training/support (coaching, webinars,**  **online modules ,**  **manuals or guidebooks, other)**  **Dosage of**

**the training (frequency**  **and length) Source of training** 

**(training provider or URL)**  **Source**

**#** Pre K educators K teachers Principals District administrators SEA administrators Other (specify)

22. When, if at all, was the KEA pilot
-tested? When, if at all, was the KEA field
-tested? What were the results?
 Pilot test:  Field test:

## Appendix F **Observation Protocol**Appendix F Observation Protocol 1 

## KEA Case Studies 

# Observation Protocol
**Note: Please stay for the whole event.** 1. Name of KEA training state: ___________________________________________

_______________ 2. Name of district and school(s) attending: _____________________________

___________________ 3. Name(s) of study team member(s): ____________________________________

_________________ 4. Date of observation: _______________________ 5. Type of training **(Q5):**  In
- person training (specify city)  We binar (provide URL if archived)  Online self
- paced module (provide URL)  Other (specify): _______________________________________________ 6. Where does this session fit into other state and district training and supports about the KEA? (What came before? What is planned to come after?) **(Q5)** 7. Name(s) of trainer and affiliation **(Q5)** : ___________________________________

_______________  Describe whether conducted by a statewide trainer or a local trainer who is part of a train
-the
trainer or another model. 8. Schedul ed training start and end time **(Q5)** : _________________________________

_____________ **Participants** 9. Who is at the training? **(Q5) Number**  K teachers Other teachers School administrators Trainers District staff Other (specify):Appendix F Observation Protocol 2

**Activities 10.**  **Enter the agenda below and describe major activities, questions, and responses**  **under each agenda item.**  **(Q5)**  [Agenda item] o Topic or learning objective o Format (lecture, small group work, role play, practice sessions) o Materials used (videos, handouts ) o Partic ipant engagement (level of engagement) o Questions asked by participants and responses provided o Amount of time  [Agenda item] o Topic or learning objective o Format (lecture, small group work, ro le play, practice sessions ) o Materials used (videos, handouts ) o Participant engagement (level of engagement) o Questions asked by participants and responses provided o Amount of time  [Agenda item] o Topic or learning objective o Format (lecture, small group work, role play, practice sessions, etc.) o Materials used (videos, hand outs, o Participant engagement (level of engagement) o Questions asked by participants and responses provided o Amount of time [Add more agenda items as needed] 

### Using your notes, complete the checklists below soon after the observation. **Specify the topic(s) tha t were addressed. Mark all that apply.**  **11. Purpose of KEA (Q1b,5)**   Screening: To identify children who may need additional supports or further assessment  Data
- informed instruction: To identify specific skills to teach to individual students or groups of students  Family engagement: To inform parents about children’s strengths and needs and ways to support learning  Professional development: To inform professional development for teachers about a common student need  Formative assessment: To benchmark learning in order to track progress throughout the year  EC program improvement: To inform improvement of early learning programs/policies  OtherAppendix F Observation Protocol 3

**12. Information and background on KEA implementation (Q1,5)**   KEA selection pro cess — Summary of how district/state KEA instrument was selected or developed  Stakeholder engagement
—The KEA selection/modification/development was informed by stakeholders  Evidence
- based, including reliability and validity  Pilot
-/field

-tested
—The KEA was fi eld
-tested and key findings presented  Cultural competence
— Meets needs of diverse study body of state/districts (e.g., poverty, race/ethnicity)  Mode of assessment
— Description of mode of assessment: checklist, observational, or direct assessment  Alignment of KEA with standards, initiative, policy, or other assessments
— early learning standards, Common Core Standards, other K
- 12 standards and assessments  Other (specify): ______________________________________________

__________________ **13. Overview of essential dom**  **ains of school readiness skills (Q2,5)**  Language and literacy development  Cognition and general knowledge (including mathematics and early scientific development)  Approaches to learning  Social and emotional development  Physical well
- being and motor development (including adaptive skills)  Other (specify): _______________________________________________

_________________ **14. KEA for special populations: English learners (EL) and children with disabilities and special needs**  **(Q5,6,6f)**  Instrument appropriate for ELs
— Information provided on appropriateness of KEA for this population  Administration accommodations provided ELs
— Specific instructions or accommodations for assessing ELs, including having instrument in multiple languages or conducting the assessment in the child’s language spoken in the home  Instrument appropriate for children with disabilities and special needs
— Information provided on appropriateness of KEA for this population  Administration accommodations for children with disabilities and special n eeds
— Specific instructions or accommodations for assessing children with disabilities and special needs  Population not appropriate for the KEA assessment
— Whether the KEA administration excludes assessment of certain groups of students (e.g., instrument not appropriate for children in certain disability categories, age year, or characteristics)Appendix F Observation Protocol 4

**15. Logistics of administration (Q5,6,6a,b,d,e,g,7)**   Administration timeline
— Indicated when KEA assessment should happen (e.g., first 2 weeks of school/enrollment, first 60 days of school/enrollment, 6 months after first administrations, end of school year)  Administration frequency
— Described the number of required administrations (e.g., beginning of school year, middle of school year, and end of school year, or at teache r’s or district’s discretion)  Optional administrations
— In addition to the required administrations, teachers are encouraged to use the KEA more frequently for progress monitoring  Personnel administering the KEA
— Description (or implied) of appropriate perso nnel to conduct the KEA (e.g., any staff who attended the appropriate trainings, teacher aids, parents)  Consent requirements — Discussed school or district policy on consent and/or process to obtain consent, if needed  Family data
— Described how to gather data from families if applicable  Data recording
— Process of providing/submitting data to school/district/state (e.g., via paper submission through schools; enter or upload data online)  Special circumstances
— Procedure for assessing children who enrolled later in the school year, children who are chronically absent, late to school or sick, etc.  Optional administration procedures
— List any mentioned: **16. Support for reliability of KEA (Q6,**  **8, 10)**   Requirements to complete the assessment
— Required ratings on specified items to obtain a valid score; required documents or documentation  Strategies to improve rating or assessment reliability  Use of **technology tools** to collect, enter or score assessments/report on results  Demonstration of how to administer the measure  Practi ce ratings (role
- play, vignette, video exercises)  Reliability check (via role
- play, vignette, or in person)  Planning for classroom implementation
— Allowed teachers to discuss strategies to incorporate KEA assessment as part of routine classroom practices  Ce rtification exam  Procedures that state or district will use to monitor implementation  Other **17. Data use and interpretation (Q10,**  **12)**  Understanding scores/results from KEA assessment  Use of scores or results from KEA for lesson planning  Use of scores or resul ts from KEA for individual instruction  Sharing individual child’s data with parents or guardians  Sharing class
-, grade
-, or school
- level data with parents  Demonstration of data use for lesson planning, sharing data with parents  Designing tasks/assignments
— teacher(s) focus on development of specific learning tasks/assignments based on KEA data in small groups, with coach, or individually  Policies regarding how KEA data should NOT be usedAppendix F Observation Protocol 5

**18. Sources for continued training and support (Q6b,**  **10)**  Online support  TA support provided by district or school  TA support provided by outside consultants  In
- person support provided by coach  Online collaboration with other teachers using the KEA  Follow
- up meetings **Summary**  What type of training is this?  Where does this session fit into other state and district training and supports about the KEA? (What came before? What is planned to come after?)  Write a brief narrative of the session, including the purpose and focus (stated or implied) and the sequence of topics and activities. Describe how the trainer/facilitator and teachers interacted.  Assess whether teachers’ questions were answered and suggested KEA processes were clearly explained. Where, if at all, did there appear to be confusion or unknowns?  W hat stood out: The topics? The strategies? The overall training? Teacher participation?  What strengths or key features may suggest this district as an illustrative case study?  Any questions or notes for special follow
- up?  Other? **Materials Collected Gather**  **materials (as available)**  Presentation and handout materials (MUST GET)  Attendance/sign

- in or registration list  Evaluation form  Evaluation results  Online materials referenced in training (or before training)  CEU paperwork List documents and materials collected: **Title**