09000064816fc32b 1 

## KEA Case Studies 

# State Version Interview Protocol **Introduction and Purpose of Interview** Thank you for your willingness to participate in this interview. We are talking with you and a few other people in your state who are knowledgeable about how your state selected or developed a Kindergarten Entry Assessment (or KEA) and how it is implementi ng it and using its results. We’d like to learn about what is working well in addition to the challenges and how you’re working to overcome them. We believe information about your experiences will be valuable for the many other states interested in impleme nting KEAs. It also has the potential to inform federal policy and to help the U.S. Department of Education develop technical assistance that is useful for all states tackling this important effort. This interview will take approximately 90 minutes. After we interview people who have been involved in decisionmaking and administration of the KEA at the state level, we will learn more about on
-the
-ground implementation and use of data from phone interviews with school district staff and in
- person interviews w ith principals and teachers. **Consent Process** Email the consent form to the respondent before the interview so he or she can read it, sign it, and send it back to you. Ask verbal permission to audio
-record the conversation. **In our first set of questions, we are trying to learn details about the [** **selection and modification or development]**  **of your KEA. We understand that your state has [ selected and modified XX measure OR**  **developed its own measure XX ]. We are interested in understanding the process you went**  **through to [** **select, modify, or develop ] the [** **measure ], such as the factors you needed to consider, stakeholders**  **that were involved, and how much time it took to reach a decision. Are you knowledgeable about the history of the [** **selection and modification] [or development]**  **process for the KEA in your state? (If not, skip to question 5.)**  1. Can you tell me a little about the history of KEAs in [state ]? Let’s start with your timeline: **(Q1a**   ** e, g, h, 2c****,8 )** 
a. When did the initiative to implement a KEA begin? b. What p rompted your state’s development or adoption of a KEA? c. About how long overall did it take to reach a decision about the [selection, modification, or development] of the [ measure ]? d. Who were the key drivers in initiating a KEA for your state (e.g., state le gislature, governor’s office, Early Learning Council, foundation)? e. Is there a particular agency, office, or committee with oversight of the selection and implementation of the KEA (e.g., Early Learning Council, governor’s office, Department of Education, Department of Health and Human Services)? f. Is conducting the KEA mandatory? How did you decide whether to make the KEA mandatory or optional? g. How, if at all, does the KEA fit in with other assessments your state is using (e.g., Comprehensive Assessment Syst em for early learning in the RTT
- ELC states)?09000064816fc32b 2 h. How, if at all, did the availability of federal funding affect planning and decisions? 2. For what purposes was the KEA selected or developed in your state? **(Q1b)** Let them first answer the open
- ended question
. In addition to the purposes you mentioned, are any of the following uses of the KEA key purposes for your state? (Mark all that are confirmed) :  Screening: To identify children who may need additional supports or further assessment  Data
- informed instructio n: To help teachers identify specific skills to teach to individual students or groups of students  Family engagement: To inform parents about children’s strengths and needs and ways to support learning  Professional development: To inform professional devel opment for teachers about a common student need  Formative assessment: To benchmark learning in order to track progress throughout the year  EC program improvement: To inform improvement of early learning programs/policies by understanding the needs of chil dren  Other (specify):

________________________________________________________________ 3. Who was involved in the selection of your state’s KEA measure and process? **(Q1a)** In addition to the groups you mentioned, were any of the following groups engaged in the selection of KEA measure and process? (Mark all that are confirmed) :  District representatives  Kindergarten teachers  PreK teachers  Parents  Foundation representatives  Outside contractors/expert consultants  Other (specify):
a. What was the range of engagement of each of these stakeholder groups? Probe : For example, some may have provided review and approval of materials, others may have been active in decisionmaking. b. Were any costs associated with including these stakeholders, such as stipends, fees, or release time? If so, what funding source supported these costs? 4. Now we’d like to ask about the final decision: What were the primary reasons you chose to [ use existing, modify existing, or develop a new measure ]? **(Q1c, d)**  Probes for modification : How did you modify the [ measure ]? Why did you feel you needed to modify the [ measure ]? Were there particular aspects of the [ measure(s )] that did not fit your population, purpose(s), or approach?  Probes for development : Why did you choos e to develop a new measure?09000064816fc32b 3 a. Did you have a set of criteria that the assessment needed to meet? If so, what were the criteria? How were they weighted? How were they used? b. How, if at all, was the match to early learning standards considered? c. How, if at all, was burden considered? d. In what ways did particular aspects of the [measure ] fit your population, purpose(s), or desired approach? e. How, if at all, was the cost considered? f. How, if at all, was a connection to a PreK and/or K
–3 assessment considered? 

### ✪ Is there a document that summarizes the selection process? If so, could you share a copy? 5. What were the most important factors in supporting the selection or development of the KEA? **(Q13)** 6. What were the biggest challenges with selecting or developing the KEA? **(Q13)** a. What solutions have you identified for addressing those challenges? 7. Was the KEA pilot
- or field
-tested? **(Q3)** If yes:
a. Can you briefly describe the pilot
- or field
-testing proces s? Probe : How many people/locations participated in the testing? What was the purpose of the pilot or field test? What did you measure? b. What did you learn from the results? c. How, if at all, were the results used to modify the content or procedures of your KEA? 

### ✪ Is there a copy of the pilot and/or field test findings you could share with us? 8. Are all the districts in the state using the KEA? **(Q2c)**
a. If not, who is opting to use the assessment? 

### ✪ Do you have a list of districts that you can provide to me? b. Are there any patterns to the participation (e.g., low
- income districts, rural districts)? c. Why do you think districts that are not participating have opted out? d. What, if any, incentives are there to use the assessment? 9. What have been the approximate overall and per
- student costs to implement the KEA so far? **(Q4)**
a. What were the approximate upfront costs of developing or selecting the KEA? b. What are the approximate costs to implement and sustain the KEA each year? c. How is the KEA funded? d. How will the KEA be s ustained in the future (beyond federal grant funding)?09000064816fc32b 4

**Now we want to learn a bit more about the [ KEA measure** **] itself —**  **the skills and abilities it assesses and what you learn from it. (Q2a)**  10. First, according to our review of documents, it appears your KEA is assessing the following domains: [list them]. Please tell me if these are incorrect or if we are missing any domains. ( Mark all that apply.)  Language and literacy development;  Cognition and general knowledge, including early mathematics and early scientific development;  Approaches toward learning;  Physical well
- being and motor development, including adaptive skills;  Social and emotional development.  Other (specify):  Other (sp ecify):  Other (specify):
a. How, if at all, does the KEA align with the domains of your state’s early learning standards? 

### ✪ Is there a document that maps the KEA to your early learning standards? If so, could you share a copy? 11. For each domain, what types o f evidence are collected as part of the assessment? **(Q2b)** Probe : For example, are knowledge, skills, and abilities assessed through observation, direct assessments, and/or parent reports? 12. How is the KEA administered? **(Q6)**
a. Who administers or provides info rmation about the student for the [measure ]? **(Q6b)** b. How, if at all, are families involved in providing information? **(Q6d)** c. To what degree is technology used to support the administration or scoring of the [ measure ]? **(Q6e) In the next series of questions, we**  **want to learn a bit more about the nuts and bolts of administering your KEA.**  13. What general information is provided to parents, teachers, and administrators about the [ KEA ] administration? **(Q5)** 

### ✪ Can you share any relevant documents?09000064816fc32b 5

14. What training or su pport do teachers and administrators receive on the administration of the KEA? Please **describe** the trainings. **(Q5, 6b)**
a. So would you say your training model uses [read the following list]? ( Mark all that apply .) **(Q5a)**  Workshops  Webinars  Coaching  Online mo dules  Other (specify): ________________________________________ b. Is the training/support offered the same for all schools and districts statewide or is it school or district specific? If different, how do trainings differ and who tailors them ( **Q6g)** c. What plans are there to continue training after the initial introduction (e.g., booster training, coaching, professional learning communities)? **(Q5)** 15. When districts administer the [ measure ], is it a standard process? Or can districts or schools adapt their administration or implementation procedures? If so, what can vary and who can choose? **(Q6g)** Probe : For example, can districts choose which measure(s) they administer? Can they sample chil dren (versus universally assessing all children in their classroom)? Can they change administration procedures (e.g., opt in or out of parent reports, home observations)? 15a. If districts use different measures or processes to collect data, does the stat e collection information on the measures or processes that districts choose to use? 

### ✪ Is there a document that summarizes what measures or processes are used by district that you can share with me? 16. How, if at all, do you and your districts monitor and evaluate the implementation of KEAs? **(Q8)**
a. What does the state do if there are issues with implementation? b. In what ways, if any, does the state modify the KEA based on information you learn from monitor ing or evaluating KEA implementation? 17. Please describe the data submission process and requirements for districts. **(Q7)**
a. How do schools and districts submit data to the state? b. What role does technology play in the process? **(Q6e)** c. Are KEA results entered in to a statewide longitudinal data system? If yes, what data are entered? If no, do they become part of another student permanent academic record?09000064816fc32b 6

**Finding one assessment that can accurately identify young children’s development status across multiple lear**  **ning domains is challenging. Ensuring that the measure is also linguistically appropriate for children whose primary language is not English adds another layer of complication, as does the**  **need to accurately assess the strengths and needs of young children with developmental delays or**  **disabilities. Within this context, we’d like to ask about your considerations of these particular populations in the**  **selection and implementation of your KEA.** 18. How, if at all, was the KEA designed or selected with English learners in mind? **(Q1c, 6f)**
a. How does implementation of the KEA differ for students who are English learners (e.g., children are assessed in their home language, use of nonverbal stimuli, observations conducted by bilingual staff or adults)? b. What data has been collected on the reliability and validity of the KEA for English Learners? c. To what degree can the [ KEA measure ] distinguish between language differences and language delays ? d. What, if any, future plans are there to make the KEA more appropriate for English learners? 19. How, if at all, was the KEA designed or selected with children with developmental delays or disabilities in mind? **(Q1c, 6f)**
a. In what ways are those who administer the KEA prepared/trained to use the [ KEA measure ] with children with developmental delays or disabilities? b. What accommodations are made and for whom? (For example, children may be given additional time to perform skills, or children with physical limitations might be ab le to demonstrate a skill verbally.) c. Are there particular disability categories for which the [ KEA measure ] is not appropriate? d. Are alternate assessments used with these children? e. In what ways are the results used for screening and referral or IEP purpos es? f. In what ways are expectations or interpretations of the findings modified or adapted for children with developmental delays or disabilities? g. To what degree do you believe the [KEA measure] accurately assesses the knowledge, skills, and abilities of chi ldren with developmental delays or disabilities? h. What data has been collected on the reliability and validity of the KEA for children with developmental delays or disabilities? i. What, if any, future plans are there to make the KEA more appropriate for child ren with developmental delays or disabilities? 20. What have been the most important factors in supporting the implementation of the KEA? **(Q14)** 21. What have been the biggest challenges with implementing the KEA? **(Q14)** a. What solutions have you identified for addressing those challenges?09000064816fc32b 7

**Now we’d like to switch gears and ask you a set of questions about how KEA results are shared and used to inform policy and practice. Let’s begin with the results of the KEAs and how peop**  **le are informed of them.**  22. Who receives the results of the KEAs (check all that apply)? Probe for any groups not mentioned**. (Q9)**   SEA administration  District administration  Principals  Kindergarten teachers  Parents  Preschool educators  Others (specify): ___

_____________________________________ 23. What information does each recipient receive about the results? **(Q9a,b, 10,11e)** Probe : Let’s take it one at a time and describe the different roles and what they receive.
a. What is the schedule for sharing KEA results? In other words, when and how often does [ each recipient checked above ] receive information? b. In what format is information shared (e.g., access to database, printed or online reports, parent conferences)? c. How is the information broken out (e.g., by student demographic characteristics, districts, schools, classrooms, or by preschool program)? i. What does the information tell you for each of the groups? d. What, if any, supporting information does [ each recipient checked above ] receive to interpret results and c onsider implications for learning supports at school and at home? e. What, if any, challenges are there with sharing information with [ each recipient checked above ]? 

### ✪ Can you share example reports? **Now let’s move to how KEA results are used to inform policy and practice.**  24. How, if at all, do state education administrators use KEA results to inform decisions about issues statewide or those concerning specific districts or preschool programs? **(Q11e,f)** Ask: For example, do they use results to make decision s about (mark all that apply):  Budget allocations?  Curriculum?  Professional development offerings?  Early childhood state policy?  K
- 12 state policy?  Other areas? (specify):
a. Which information is most useful? b. What other information would have been useful? c. [Ask only if districts are allowed to choose between measures or processes ] Does the state compare results across districts using different measures or data collection processes to inform KEA implementation?09000064816fc32b 8

25. What training or support do district administrato rs receive to help them use KEA results to inform teaching and learning? **(Q10)** 26. How do you expect districts to use results? For example, do you expect them to use results to make decisions about [read list]? ( Mark all that apply) **(Q11d) :**   Budget allocation s  Curriculum  Professional development offerings  District
-wide planning for Kindergarten  Identify changes needed across PreK programs throughout the district  Other areas a. How do you support districts to use the results in this way? 27. Describe any policies regarding how KEA data should not be used (e.g., determine promotion or prevent kindergarten entry in any way). **(Q12)**
a. How, if at all, are these policies communicated to districts, schools, and teachers? b. How, if at all, are these pol icies enforced? Please give me a specific example. 28. What have been the most important factors in supporting use of KEA results to improve students’ educational experience? **(Q15)** 29. What have been the biggest challenges with using KEA results? **(Q15)** a. What solu tions have you identified for addressing those challenges? **This is our final question! In closing we’d like you to reflect on how your state, districts, and schools administered and used KEA data and try to offer lessons learned.**  30. Are there lessons lea rned or recommendations you can share with other states about KEA selection, implementation, and use of KEA results to inform policy and practice? **(Q13, 14, 15) Thank you for your time and thoughtful responses.**